Remote support system, terminal device, and remote device

ABSTRACT

A remote support system includes: a terminal device; and a remote device, the terminal device including a first processor configured to: transmit, to the remote device, image information obtained by imaging a space including a target object; receive display information that is derived by the remote device in accordance with the image information and three-dimensional position information of an item included in the space, the display information indicating a display position and an orientation of a guidance image to be virtually displayed in a three-dimensional space in an imaging area for the target object; and display the guidance image by using the display information that is received, the remote device including a second processor configured to: receive the image information from the terminal device; derive the display information in accordance with the image information that is received and the three-dimensional position information; and transmit, to the terminal device, the display information that is derived.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims priority under 35 USC 119 fromJapanese Patent Application No. 2021-132906 filed Aug. 17, 2021.

BACKGROUND (i) Technical Field

The present disclosure relates to a remote support system, a terminaldevice, and a remote device.

(ii) Related Art

Japanese Unexamined Patent Application Publication No. 2017-068689discloses a guidance support method directed at guiding a user smoothlyto a target. According to the guidance support method, targetinformation is detected from among pieces of space information stored inadvance, and the position of a display device in a space defined by thespace information and the position of the target defined by the targetinformation are measured. In addition, according to the guidance supportmethod, on the basis of the position of the display device and theposition of the target, which have been measured, the position of anicon to be displayed on the display device and the movement of the iconare determined, and the icon is displayed with the determined movementand at the determined position on the display device. Furthermore,according to the guidance support method, in accordance with a change inthe position of the display device, the position and the movement of theicon to be displayed on the display device are changed.

In addition, International Publication No. 2017/056631 discloses aninformation processing system that controls display of information to beinput by using a reception device of a captured image. The informationprocessing system includes a control unit that controls display of anintegrated image, the integrated image being obtained by integratingfirst information and second information. The first information isgenerated in accordance with an input to a first system that receivesthe captured image obtained by an image capturing system. The secondinformation is generated in accordance with an input to a second systemthat receives the captured image obtained by the image capturing system.

Furthermore, International Publication No. 2017/158718 discloses aremote operation support device including an on-site terminal having animaging unit that captures a video seen from an operator and aninstruction terminal that transmits and receives information to and fromthe on-site terminal. The instruction terminal in the remote operationsupport device includes a position-and-direction estimating unit and anon-site condition image generating unit. The position-and-directionestimating unit estimates the position and direction of the operatorfrom the video captured by the imaging unit. The on-site condition imagegenerating unit generates an image indicating an on-site conditionincluding the position of the operator from results of the estimation bythe position-and-direction estimating unit. The instruction terminal inthe remote operation support device further includes an instruction-sidedisplay unit, an operation instruction receiving unit, and a directioncalculating unit. The instruction-side display unit displays a screenincluding the image generated by the on-site condition image generatingunit. The operation instruction receiving unit receives informationindicating the next operation position that is input by an operationinstructor on the screen displayed by the instruction-side display unit.The direction calculating unit calculates the direction to the nextoperation position from results of the estimation by theposition-and-direction estimating unit and results of the reception bythe operation instruction receiving unit. Furthermore, the on-siteterminal in the remote operation support device includes a guidanceimage generating unit and an on-site-side display unit. The guidanceimage generating unit generates an image indicating the direction to thenext operation position from results of the calculation by the directioncalculating unit. The on-site-side display unit displays a screenincluding the image generated by the guidance image generating unit.

SUMMARY

Aspects of non-limiting embodiments of the present disclosure relate toa remote support system, a terminal device, and a remote device that mayguide a user to a required imaging position and a required imagingorientation more accurately than in a case where an instruction for animaging direction is not provided.

Aspects of certain non-limiting embodiments of the present disclosureaddress the above advantages and/or other advantages not describedabove. However, aspects of the non-limiting embodiments are not requiredto address the advantages described above, and aspects of thenon-limiting embodiments of the present disclosure may not addressadvantages described above.

According to an aspect of the present disclosure, there is provided aremote support system including: a terminal device; and a remote device,the terminal device including a first processor configured to: transmit,to the remote device, image information obtained by imaging a spaceincluding a target object; receive display information that is derivedby the remote device in accordance with the image information andthree-dimensional position information of an item included in the space,the display information indicating a display position and an orientationof a guidance image to be virtually displayed in a three-dimensionalspace in an imaging area for the target object; and display the guidanceimage by using the display information that is received, the remotedevice including a second processor configured to: receive the imageinformation from the terminal device; derive the display information inaccordance with the image information that is received and thethree-dimensional position information; and transmit, to the terminaldevice, the display information that is derived.

BRIEF DESCRIPTION OF THE DRAWINGS

An exemplary embodiment of the present disclosure will be described indetail based on the following figures, wherein:

FIG. 1 is a block diagram illustrating an example of a hardwareconfiguration of a remote support system according to the exemplaryembodiment;

FIG. 2 is a block diagram illustrating an example of a functionalconfiguration of the remote support system according to the exemplaryembodiment;

FIG. 3 illustrates a perspective view of an example of a guidance imageaccording to the exemplary embodiment;

FIG. 4 is a flowchart illustrating an example of a guidance imagedisplay process according to the exemplary embodiment;

FIG. 5 illustrates a front view of an example of the guidance imagedisplay screen according to the exemplary embodiment;

FIG. 6 illustrates a front view of another example of the guidance imagedisplay screen according to the exemplary embodiment;

FIG. 7 is a flowchart illustrating an example of a remote supportprocess according to the exemplary embodiment;

FIG. 8 illustrates a front view of an example of a guidance imagesetting screen according to the exemplary embodiment;

FIG. 9 illustrates a front view of another example of the guidance imagesetting screen according to the exemplary embodiment;

FIG. 10 illustrates a perspective view of another example of theguidance image according to the exemplary embodiment;

FIG. 11 illustrates a perspective view of another example of theguidance image according to the exemplary embodiment;

FIG. 12 illustrates a perspective view of another example of theguidance image according to the exemplary embodiment;

FIG. 13 illustrates a perspective view of another example of theguidance image according to the exemplary embodiment; and

FIG. 14 illustrates a front view of another example of the guidanceimage display screen according to the exemplary embodiment.

DETAILED DESCRIPTION

Now, an exemplary embodiment of the present disclosure will be describedbelow in detail with reference to the drawings. Note that the exemplaryembodiment will illustrate a case where the present disclosure isapplied to a remote support system including plural terminal devices anda remote device. Each of the terminal devices is a mobile terminal usedindividually by a corresponding user. The remote device is provided at aplace that is remote from the terminal devices. In addition, theexemplary embodiment will illustrate a case where the user is anoperator who maintains an image forming device such as a digitalmultifunction device or a printer and where the remote support system isa system that remotely supports the maintenance of the image formingdevice by the operator.

First, configurations of a remote support system 90 according to theexemplary embodiment will be described with reference to FIGS. 1 and 2 .FIG. 1 is a block diagram illustrating an example of a hardwareconfiguration of the remote support system 90 according to the exemplaryembodiment. FIG. 2 is a block diagram illustrating an example of afunctional configuration of the remote support system 90 according tothe exemplary embodiment.

As illustrated in FIG. 1 , the remote support system 90 according to theexemplary embodiment includes a remote device 10 and plural terminaldevices 30. The remote device 10 and each of the terminal devices 30 arecapable of accessing a network 80. Note that examples of the remotedevice 10 include an information processing device such as a personalcomputer or a server computer. Examples of the terminal device 30include a mobile terminal such as a smartphone, a tablet terminal, or apersonal digital assistant (PDA, mobile information terminal).

The terminal device 30 according to the exemplary embodiment is a deviceowned by a corresponding one of plural users who use the remote supportsystem 90 (hereinafter simply referred to as “user”) when maintainingthe image forming device. The terminal device 30 includes a centralprocessing unit (CPU) 31, a memory 32 as a temporary storage area, anon-volatile storage unit 33, an input unit 34 that is a touch panel orthe like, a display unit 35 that is a liquid crystal display or thelike, and a medium reader/writer (R/W) 36. The terminal device 30further includes an imaging unit 38, a microphone 39, a positiondetecting unit 40, a space information detecting unit 41, and a wirelesscommunication unit 42. The CPU 31, the memory 32, the storage unit 33,the input unit 34, the display unit 35, the medium reader/writer 36, theimaging unit 38, the microphone 39, the position detecting unit 40, thespace information detecting unit 41, and the wireless communication unit42 are connected to one another via a bus B1. The medium reader/writer36 reads information written on a recording medium 37 and writesinformation to the recording medium 37.

The storage unit 33 is implemented by a hard disk drive (HDD), a solidstate drive (SSD), a flash memory, or the like. The storage unit 33 as astorage medium stores a guidance image display program 33A. Therecording medium 37, on which the guidance image display program 33A iswritten, is set in the medium reader/writer 36, and the mediumreader/writer 36 reads the guidance image display program 33A from therecording medium 37, and thereby the guidance image display program 33Ais stored in the storage unit 33. The CPU 31 reads the guidance imagedisplay program 33A from the storage unit 33, loads the guidance imagedisplay program 33A into the memory 32, and sequentially performsprocesses included in the guidance image display program 33A.

The imaging unit 38 according to the exemplary embodiment images theimage forming device that is a maintenance target and outputs imageinformation obtained by the imaging. The microphone 39 according to theexemplary embodiment collects a voice of the user and outputs voiceinformation.

The position detecting unit 40 according to the exemplary embodimentdetects the position of the terminal device 30 and outputs positioninformation. In the exemplary embodiment, the position detecting unit 40uses the global positioning system (GPS). However, the presentdisclosure is not limited to this exemplary embodiment. For example, theposition detecting unit 40 may use position information acquired from aWi-Fi (registered trademark) router, may detect the position by using abeacon, or may detect the position through image analysis using acaptured image.

The space information detecting unit 41 according to the exemplaryembodiment detects information indicating a three-dimensional positionof an item included in a space in front of the terminal device 30(hereinafter referred to as “three-dimensional position information”).The space information detecting unit 41 according to the exemplaryembodiment detects the three-dimensional position information in a spacecorresponding to an imaging angle of view of the imaging unit 38. In theexemplary embodiment, a depth sensor is used as the space informationdetecting unit 41. However, the present disclosure is not limited tothis exemplary embodiment. For example, a three-dimensional scanner orthe like may also be used as the space information detecting unit 41.

The remote device 10, on the other hand, is a device that plays a majorrole in the remote support system 90 and that derives a guidance image,which will be described later in detail, to be presented to the user.The remote device 10 includes a CPU 11, a memory 12 as a temporarystorage area, a non-volatile storage unit 13, an input unit 14 such as akeyboard or a mouse, a display unit 15 such as a liquid crystal display,a medium reader/writer 16, and a communication interface (I/F) unit 18.The CPU 11, the memory 12, the storage unit 13, the input unit 14, thedisplay unit 15, the medium reader/writer 16, and the communication I/Funit 18 are connected to one another via a bus B2. The mediumreader/writer 16 reads information written on a recording medium 17 andwrites information to the recording medium 17.

The storage unit 13 is implemented by an HDD, an SSD, a flash memory, orthe like. The storage unit 13 as a storage medium stores a remotesupport program 13A. The recording medium 17, on which the remotesupport program 13A is written, is set in the medium reader/writer 16,and the medium reader/writer 16 reads the remote support program 13Afrom the recording medium 17, and thereby the remote support program 13Ais stored in the storage unit 13. The CPU 11 reads the remote supportprogram 13A from the storage unit 13, loads the remote support program13A into the memory 12, and sequentially performs processes included inthe remote support program 13A.

Note that a public communication line such as the Internet or atelephone network is used as the network 80 in the exemplary embodiment.However, the present disclosure is not limited to this exemplaryembodiment. For example, a communication line in a company such as alocal area network (LAN) or a wide area network (WAN) may be used as thenetwork 80, or a combination of such a communication line in a companyand a public communication line may be used as the network 80.

Next, functional configurations of the remote device 10 and the terminaldevice 30 according to the exemplary embodiment will be described withreference to FIG. 2 .

As illustrated in FIG. 2 , the terminal device 30 according to theexemplary embodiment includes a first transmitting unit 31A, a firstreceiving unit 31B, and a first display control unit 31C. By executingthe guidance image display program 33A, the CPU 31 of the terminaldevice 30 functions as the first transmitting unit 31A, the firstreceiving unit 31B, and the first display control unit 31C.

The first transmitting unit 31A according to the exemplary embodimenttransmits, to the remote device 10, image information obtained byimaging a space including a target object (hereinafter also referred toas “image information”). Note that the image forming device that is amaintenance target is used as the target object in the exemplaryembodiment, as described above.

The first receiving unit 31B according to the exemplary embodimentreceives display information that is derived by the remote device 10 inaccordance with the image information and three-dimensional positioninformation of an item included in the above space (hereinafter simplyreferred to as “display information”), the display informationindicating a display position and an orientation of a guidance image tobe virtually displayed in a three-dimensional space in an imaging areafor the target object. Note that the first transmitting unit 31Atransmits the image information, and the first receiving unit 31Breceives the display information via the wireless communication unit 42in the exemplary embodiment. However, the present disclosure is notlimited to this exemplary embodiment.

The first display control unit 31C according to the exemplary embodimentcauses the display unit 35 to display the guidance image by using thedisplay information that is received.

As illustrated in FIG. 2 , the remote device 10 according to theexemplary embodiment, on the other hand, includes a second receivingunit 11A, a second display control unit 11B, a deriving unit 11C, and asecond transmitting unit 11D. By executing the remote support program13A, the CPU 11 of the remote device 10 functions as the secondreceiving unit 11A, the second display control unit 11B, the derivingunit 11C, and the second transmitting unit 11D.

The second receiving unit 11A according to the exemplary embodimentreceives the image information from the terminal device 30. The derivingunit 11C according to the exemplary embodiment derives the above displayinformation in accordance with the image information that is receivedand the three-dimensional position information that is described above.Furthermore, the second transmitting unit 11D according to the exemplaryembodiment transmits, to the terminal device 30, the display informationthat is derived.

Note that, in the remote support system 90 according to the exemplaryembodiment, the second display control unit 11B causes the display unit15 to display a guidance image setting screen, which will be describedlater in detail, by using the image information that is received and thethree-dimensional position information that is described above. Thederiving unit 11C according to the exemplary embodiment causes a personwho uses the remote device 10 to set, on the guidance image settingscreen displayed on the display unit 15, the display position and theorientation of the guidance image to derive the display information.

In the exemplary embodiment, image information indicating a still imageis used as the image information. Thus, the imaging unit 38 provided inthe terminal device 30 is capable of capturing a still image. However,the present disclosure is not limited to this exemplary embodiment. Forexample, it is needless to say that an imaging unit capable of capturingboth a still image and a moving image may also be used as the imagingunit 38. In addition, in the exemplary embodiment, image informationindicating a color image is used as the image information, and thus, theimaging unit 38 according to the exemplary embodiment is capable ofcapturing a color image. However, the present disclosure is not limitedto this exemplary embodiment. For example, image information indicatinga monochrome image may also be used as the image information, and animaging unit capable of capturing a monochrome image may also be used asthe imaging unit 38.

In addition, in the exemplary embodiment, the three-dimensional positioninformation to be used by the remote device 10 is obtained by the spaceinformation detecting unit 41 of the terminal device 30, transmitted bythe first transmitting unit 31A thereof, and received by the secondreceiving unit 11A of the remote device 10. However, the presentdisclosure is not limited to this exemplary embodiment, andthree-dimensional position information may be separately obtained byusing measuring equipment different from the space information detectingunit 41 of the terminal device 30 and may be stored in the storage unit13 of the remote device 10 in advance.

In addition, in the exemplary embodiment, the guidance image has aninstruction plane that is virtually displayed at the above displayposition and with the above orientation in the above three-dimensionalspace. In particular, the guidance image according to the exemplaryembodiment is a polyhedron having the instruction plane. Furthermore,the polyhedron according to the exemplary embodiment is a regularpolyhedron. FIG. 3 illustrates a perspective view of an example of aguidance image 60 according to the exemplary embodiment.

As illustrated in FIG. 3 , the guidance image 60 according to theexemplary embodiment is a polyhedron 60B having an instruction plane 60Aas the above-described instruction plane, and the polyhedron 60B is aregular hexahedron.

The instruction plane 60A of the guidance image 60 according to theexemplary embodiment is a plane that is inside the polyhedron 60B andthat has, as vertices, some of vertices 60 a to 60 h of the polyhedron60B (vertices 60 a, 60 c, and 60 f in the example illustrated in FIG. 3). Thus, in this case, the instruction plane 60A is a regular trianglewhen viewed in the direction facing the instruction plane 60A.

In the above manner, in the exemplary embodiment, a guidance imageincluding the polyhedron 60B is used as the guidance image. However, thepresent disclosure is not limited to this exemplary embodiment. Forexample, the instruction plane 60A may also be used alone as a guidanceimage not including the polyhedron 60B.

Furthermore, the first display control unit 31C according to theexemplary embodiment changes a display state of the guidance image 60 inaccordance with a deviation amount between an actual imaging state ofthe terminal device 30 and an imaging state corresponding to the abovedisplay position and the above orientation. Here, in the exemplaryembodiment, a transmittance is used as the above display state and isincreased as the above deviation amount decreases. However, the presentdisclosure is not limited to this exemplary embodiment, and, instead ofthe transmittance, any one of a blinking interval and a display colormay also be used as the above display state, or a combination of two orthree of the blinking interval, the display color, and the transmittancemay also be used as the display state.

In addition, in the exemplary embodiment, when the first display controlunit 31C images a target position of the target object, a marker isvirtually displayed at the target position.

Next, operations of the remote support system 90 according to theexemplary embodiment will be described with reference to FIGS. 4 to 9 .First, operations of the terminal device 30 according to the exemplaryembodiment will be described with reference to FIGS. 4 to 6 . FIG. 4 isa flowchart illustrating an example of a guidance image display processaccording to the exemplary embodiment. Note that a process for providingan instruction regarding a target operation of remote support by theremote support system 90 will be omitted from the following descriptionto avoid complexity, and a process for guiding a user to a requiredimaging position and a required imaging orientation will be describedbelow.

In the remote support system 90 according to the exemplary embodiment,the user images a space including a target object in a predetermineddirection (toward the front surface of the target object in theexemplary embodiment) by using the terminal device 30, and imageinformation obtained by the imaging unit 38 in response to the imagingand three-dimensional position information obtained by the spaceinformation detecting unit 41 at the same timing as the imaging arestored in the storage unit 33.

When the user inputs an instruction for executing the guidance imagedisplay process to the terminal device 30 in this state, the CPU 31 ofthe terminal device 30 executes the guidance image display program 33Ato execute the guidance image display process illustrated in FIG. 4 .

In step 100 in FIG. 4 , the CPU 31 reads the image information and thethree-dimensional position information from the storage unit 33. In step102, the CPU 31 transmits, to the remote device 10, the imageinformation and the three-dimensional position information that areacquired.

Although details will be described later, upon reception of the imageinformation and the three-dimensional position information, the remotedevice 10 derives the above-described display information (hereinafterreferred to as “guidance image display information”) by using the imageinformation and the three-dimensional position information. The guidanceimage display information is information for displaying the guidanceimage 60 for guiding the user to the required imaging position and therequired imaging orientation. Subsequently, the remote device 10transmits the derived guidance image display information to the terminaldevice 30 that is an access source. At this time, the remote device 10transmits, to the terminal device 30, together with the guidance imagedisplay information, marker display information for virtually displayinga marker at a target position for maintenance of a target object.

In step 104, the CPU 31 waits until receiving the guidance image displayinformation and the marker display information from the remote device10.

In step 106, the CPU 31 determines each of an imaging position and anorientation (equivalent to imaging direction) of the terminal device 30at this time point. Here, information indicating the above imagingposition is obtained from the position detecting unit 40. In addition,information indicating the above orientation is obtained by a gyrosensor(omitted from illustration) incorporated in the terminal device 30.However, the present disclosure is not limited to this exemplaryembodiment. For example, information indicating the above orientationmay also be obtained by an acceleration sensor or the like, instead ofthe gyrosensor.

In step 108, the CPU 31 controls the display unit 35 to display aguidance image display screen having a predetermined configuration byusing the guidance image display information and the marker displayinformation that are received, the imaging position and the orientationof the terminal device 30 that are determined, and an image that iscaptured by the imaging unit 38 at this time point (hereinafter referredto as “actual captured image”). FIG. 5 illustrates an example of theguidance image display screen according to the exemplary embodiment.

As illustrated in FIG. 5 , the guidance image display screen accordingto the exemplary embodiment displays a message for encouraging the userto set the imaging position and the orientation of the terminal device30 so as to face the instruction plane 60A in the polyhedron 60B in theguidance image 60. In addition, on the guidance image display screenaccording to the exemplary embodiment, the guidance image 60 isdisplayed by being superimposed on the actual captured image such thatthe instruction plane 60A corresponds to the display position and theorientation indicated by the guidance image display information.Furthermore, on the guidance image display screen according to theexemplary embodiment, in accordance with a deviation amount between anactual imaging state and an imaging state corresponding to the requiredimaging position and the required imaging orientation, the display state(transmittance in the exemplary embodiment) of the guidance image 60 issequentially changed (increased as the deviation amount decreases in theexemplary embodiment).

That is to say, the guidance image 60 according to the exemplaryembodiment is a three-dimensional image in which the instruction plane60A becomes a regular triangle when the imaging position and theorientation of the terminal device 30 correspond to the required imagingposition and the required imaging orientation.

When the imaging position and the orientation of the terminal device 30differ from the required imaging position and the required imagingorientation, the guidance image 60 in a case where the guidance image60, in which the instruction plane 60A becomes a regular triangle withthe required imaging position and the required imaging orientation, isviewed in this state is displayed at the imaging position on theterminal device 30 according to the exemplary embodiment.

The example illustrated in FIG. 5 is an example of the guidance imagedisplay screen in a case where a target object 50 is imaged to face thefront surface thereof at this time point and where the required imagingposition faces the left side surface of the target object 50. In thiscase, the instruction plane 60A of the guidance image 60 is distortedfrom a regular triangle, and the size thereof decreases as the distancefrom the terminal device 30 to the required imaging position becomeslonger. Accordingly, by referring to the guidance image display screen,the user is able to intuitively grasp the required imaging position andthe required imaging orientation for the target object 50, on the basisof the display position and the size of the guidance image 60 and theshape of the instruction plane 60A.

The example illustrated in FIG. 6 is an example of the guidance imagedisplay screen in a case where the terminal device 30 becomes closer tothe required imaging position and the required imaging orientation fromthe example illustrated in FIG. 5 . In this case, the target object 50is displayed in a state of being imaged to face the left side surfacethereof, and the instruction plane 60A of the guidance image 60 becomesalmost a regular triangle. In addition, since the target position isincluded in the imaging angle of view in this case, a marker 70 as theabove-described marker is displayed. Accordingly, the user is also ableto grasp the target position.

Note that the terminal device 30 sequentially generates informationindicating the guidance image 60 having a different display position,shape, size, and transmittance depending on the imaging position and theorientation of the terminal device 30 in the exemplary embodiment.However, the present disclosure is not limited to this exemplaryembodiment. For example, the remote device 10 may sequentially generateinformation indicating the guidance image 60 and may transmit theinformation to the terminal device 30.

In step 110, the CPU 31 determines whether the terminal device 30 is atthe required imaging position and with the required imaging orientationand guiding of the user to a destination is completed. If thedetermination is negative, the process returns to step 106; if thedetermination is positive, the guidance image display process ends.

Through a repeated process of steps 106 to 110, the display unit 35 ofthe terminal device 30 displays the target object 50 imaged by theterminal device 30, and also displays the guidance image 60 in asuperimposed manner in accordance with the imaging position and theorientation of the terminal device 30. Accordingly, by referring to thedisplay screen, the user is able to intuitively move the terminal device30 to the required imaging position and the required imagingorientation.

Upon ending of the guidance image display process, a remote supportprocess for expected maintenance of the target object 50 is executed byusing a technique of the known related art.

Next, operations of the remote device 10 according to the exemplaryembodiment will be described with reference to FIGS. 7 to 9 . FIG. 7 isa flowchart illustrating an example of a remote support processaccording to the exemplary embodiment. Note that a case where the useris remotely instructed by a person who uses the remote device 10(hereinafter also referred to as “remote instructor”) will be describedhere.

In the remote support system 90 according to the exemplary embodiment,the remote support process illustrated in FIG. 7 is executed by the CPU11 of the remote device 10 starting execution of the remote supportprogram 13A at each predetermined period (every 10 minutes in theexemplary embodiment) in available hours (hours from 8:00 am to 5:00 pmin the exemplary embodiment) on available days (weekdays in theexemplary embodiment) of the remote support system 90 by the user.

In step 200 in FIG. 7 , the CPU 11 waits until receiving the imageinformation and the three-dimensional position information from any ofthe terminal devices 30. In step 202, the CPU 11 controls the displayunit 15 to display the guidance image setting screen having apredetermined configuration by using the image information and thethree-dimensional position information that are received.

Subsequently, in step 204, the CPU 11 waits for an input ofpredetermined information. FIG. 8 illustrates an example of the guidanceimage setting screen according to the exemplary embodiment.

As illustrated in FIG. 8 , a message for encouraging the remoteinstructor to designate the position of the polyhedron and theinstruction plane and a message for encouraging the remote instructor todesignate the above-described target position are displayed on theguidance image setting screen according to the exemplary embodiment. Inaddition, by using a captured image indicated by the received imageinformation and a three-dimensional position of each item indicated bythe received three-dimensional position information, thethree-dimensional space in the imaging area corresponding to thecaptured image is virtually displayed in a state of including the targetobject 50 on the guidance image setting screen according to theexemplary embodiment. Furthermore, the polyhedron 60B in the guidanceimage 60 is displayed at a predetermined default position and with adefault size on the guidance image setting screen according to theexemplary embodiment.

Accordingly, by referring to the guidance image setting screen, theremote instructor is able to grasp, as a position in abovethree-dimensional space, the position of the target object 50 imaged bythe user of the access source (hereinafter simply referred to as “user”)by using the terminal device 30. Thus, the remote instructor moves thedisplayed polyhedron 60B to the required imaging position (the positionof the user at the start of maintenance of the target object 50 in theexemplary embodiment) via the input unit 14. At this time, the CPU 11 ofthe remote device 10 sequentially changes the size of the polyhedron 60Bin accordance with the distance between the position of the polyhedron60B that is being moved by the remote instructor in the abovethree-dimensional space and the imaging position of the terminal device30. Accordingly, the remote instructor is able to grasp the position ofthe polyhedron 60B in the front-back direction on the basis of the sizeof the polyhedron 60B.

Upon completion of moving the polyhedron 60B to the required imagingposition, the remote instructor designates vertices of the polyhedron60B that constitutes an instruction plane for which the orientation(tilt angle) faces the required imaging direction or a direction that isthe closest to the required imaging direction, via the input unit 14. Inaccordance with the designation, the guidance image 60 in the stateillustrated in FIG. 9 , for example, is displayed on the guidance imagesetting screen.

In the above manner, in the exemplary embodiment, as an operationrelated to the guidance image 60, the polyhedron 60B is moved to therequired imaging position, and then, the instruction plane 60A isdesignated. However, the present disclosure is not limited to thisexemplary embodiment. For example, first, the instruction plane 60A maybe designated, and then, the guidance image 60 may be moved.

The remote instructor also designates the above-described targetposition in the target object 50 via the input unit 14. At this time, insome cases, the target position in the target object 50 is not displayedon the displayed guidance image setting screen. In such a case, thetarget object 50 or the guidance image setting screen is rotated in thethree-dimensional space such that the target position is seen, and then,the target position is designated.

Upon completion of the above designation related to the guidance image60 and designation of the target position, the remote instructor selectsan end button 15A via the input unit 14. In response to this, thedetermination in step 204 is positive, and the process advances to step206.

In step 206, by using pieces of information designated by the remoteinstructor on the guidance image setting screen, the CPU 11 derives theguidance image display information and the marker display informationthat are described above. In step 208, the CPU 11 transmits, to theterminal device 30 that is the access source, the guidance image displayinformation and the marker display information that are derived, andthen ends the remote support process.

Note that the above exemplary embodiment has illustrated a case where atransmittance is used as the display state of the guidance image, whichis changed in accordance with the deviation amount between the actualimaging state and the imaging state corresponding to the requiredimaging position and the required imaging orientation. However, thepresent disclosure is not limited to this exemplary embodiment. Forexample, at least one of a blinking interval and a display color mayalso be used as the above display state, or a combination of pluraltypes among three types of display states including the blinkinginterval, the display color, and the transmittance may also be used asthe display state, as described above. For example, in a case where ablinking interval is used as the display state, the blinking intervalmay be made shorter as the deviation amount decreases; in a case where adisplay color is used as the display state, the display color is madecloser to red as the deviation amount decreases.

In addition, the above exemplary embodiment has illustrated a case wherethe instruction plane 60A having a tilt angle of 45 degrees, asillustrated in FIG. 3 for example, is used as the instruction plane ofthe guidance image 60. However, the present disclosure is not limited tothis exemplary embodiment. For example, a perpendicular plane may alsobe used as the instruction plane 60A.

In this case, when a diagonally forward right direction is wished to bedesignated as the imaging direction, as illustrated in FIG. 10 forexample, four vertices, which are the vertex 60 a, the vertex 60 c, thevertex 60 e, and the vertex 60 g, may be designated. In addition, whenan outer circumferential surface of the polyhedron 60B is wished to bedesignated as the instruction plane 60A, for example, the vertex 60 b,the vertex 60 c, the vertex 60 f, and the vertex 60 g, may bedesignated. In this case, instead of designating the four vertices, aplane that is wished to be the instruction plane 60A may also bedirectly designated. In this exemplary embodiment, since the instructionplane 60A may be designated by a single designating operation, theinstruction plane 60A may be designated more easily than in a case wherethe instruction plane 60A is designated by designating the vertices.

In addition, the above exemplary embodiment has illustrated a case wherea regular hexahedron as a polyhedron is used in the guidance image.However, the present disclosure is not limited to this exemplaryembodiment. As illustrated in FIG. 11 for example, a guidance image 64in which the polyhedron is a regular tetrahedron may also be used, andalso, a hexahedron, a tetrahedron, or the like whose sides havedifferent lengths may also be used. Furthermore, the guidance imageusing a polyhedron is not limited to the above one having theinstruction plane 60A alone. As illustrated in FIG. 12 for example, aguidance image 66 including an instruction plane and further including acolumnar image 66A reaching a target object may also be used. Note thatthe instruction plane and the target object are omitted from theillustration in FIG. 12 to avoid complexity.

Furthermore, as illustrated in FIG. 13 for example, in addition to theguidance image 60, the display unit 35 of the terminal device 30 mayalso display plural polyhedrons 60B in a juxtaposed manner to includethe area including the target object, the plural polyhedrons having ashape and a size identical to the shape and the size of the guidanceimage 60 and not having the instruction plane 60A. According to thisexemplary embodiment, even when the target object 50 is presentcomparatively away, the required imaging position and the requiredimaging orientation that are designated by the remote instructor areeasy to grasp.

Although not referred to in the above exemplary embodiment, asillustrated in FIG. 14 for example, when the actual imaging statecorresponds to required imaging position and the required imagingorientation, under control of the first display control unit 31C, a mark60C having a shape and a size identical to the shape and the size of theinstruction plane 60A may be further displayed. Note that in this case,when the instruction plane 60A corresponds to the mark 60C, informationindicating the completion of guiding of the user may be automaticallytransmitted to the remote device 10.

Although the exemplary embodiment has been described above, thetechnical scope of the present disclosure is not limited to the scopedescribed in the above exemplary embodiment. Various modifications andalternations may be added to the above exemplary embodiment withoutdeparting from the spirit of the disclosure, and embodiments with themodifications and alternations are also included in the technical scopeof the present disclosure.

The above exemplary embodiment is not intended to limit the disclosuredefined by the claims, and not all of the combinations of featuresdescribed in the exemplary embodiment are expected to be used in thedisclosure. The exemplary embodiment described above includes examplesin various stages, and various examples are extracted in accordance withcombinations of plural disclosed elements. If some elements of all theelements described in the exemplary embodiment are removed, theconfiguration with these elements removed may be extracted as an exampleas long as an effect is obtained.

In the embodiments above, the term “processor” refers to hardware in abroad sense. Examples of the processor include general processors (e.g.,CPU: Central Processing Unit) and dedicated processors (e.g., GPU:Graphics Processing Unit, ASIC: Application Specific Integrated Circuit,FPGA: Field Programmable Gate Array, and programmable logic device).

In the embodiments above, the term “processor” is broad enough toencompass one processor or plural processors in collaboration which arelocated physically apart from each other but may work cooperatively. Theorder of operations of the processor is not limited to one described inthe embodiments above, and may be changed.

Furthermore, the above exemplary embodiment has illustrated a case wherethe guidance image display process and the remote support process areimplemented by software configurations using a computer by executingprograms. However, the present disclosure is not limited to thisexemplary embodiment. For example, the guidance image display processand the remote support process may be implemented by hardwareconfigurations or a combination of a hardware configuration and asoftware configuration.

Furthermore, the configurations of the remote device 10 and the terminaldevice 30 described in the above exemplary embodiment are examples, andthus, it is needless to say that an unnecessary part may be omitted or anew part may be added without departing from the spirit of the presentdisclosure.

In addition, the flows of the guidance image display process and theremote support process described in the above exemplary embodiment arealso examples, and thus, it is needless to say that an unnecessary stepmay be skipped, a new step may be added, or the order in the process maybe changed, without departing from the spirit of the present disclosure.

What is claimed is:
 1. A remote support system comprising: a terminaldevice; and a remote device, the terminal device including a firstprocessor configured to: transmit, to the remote device, imageinformation obtained by imaging a space including a target object;receive display information that is derived by the remote device inaccordance with the image information and three-dimensional positioninformation of an item included in the space, the display informationindicating a display position and an orientation of a guidance image tobe virtually displayed in a three-dimensional space in an imaging areafor the target object; and display the guidance image by using thedisplay information that is received, the remote device including asecond processor configured to: receive the image information from theterminal device; derive the display information in accordance with theimage information that is received and the three-dimensional positioninformation; and transmit, to the terminal device, the displayinformation that is derived.
 2. The remote support system according toclaim 1, wherein the first processor is further configured to transmitthe three-dimensional position information to the terminal device. 3.The remote support system according to claim 1, wherein the guidanceimage has an instruction plane that is virtually displayed at thedisplay position and with the orientation in the three-dimensionalspace.
 4. The remote support system according to claim 2, wherein theguidance image has an instruction plane that is virtually displayed atthe display position and with the orientation in the three-dimensionalspace.
 5. The remote support system according to claim 3, wherein theguidance image is a polyhedron having the instruction plane.
 6. Theremote support system according to claim 4, wherein the guidance imageis a polyhedron having the instruction plane.
 7. The remote supportsystem according to claim 5, wherein the polyhedron is a regularpolyhedron.
 8. The remote support system according to claim 6, whereinthe polyhedron is a regular polyhedron.
 9. The remote support systemaccording to claim 5, wherein the instruction plane is a plane that isinside the polyhedron and that has, as vertices, some of vertices of thepolyhedron.
 10. The remote support system according to claim 6, whereinthe instruction plane is a plane that is inside the polyhedron and thathas, as vertices, some of vertices of the polyhedron.
 11. The remotesupport system according to claim 5, wherein the first processor isfurther configured to display, in addition to the guidance image, aplurality of polyhedrons in a juxtaposed manner with respect to an areaincluding the target object, the plurality of polyhedrons having a shapeand a size identical to a shape and a size of the polyhedron and nothaving the instruction plane.
 12. The remote support system according toclaim 5, wherein the first processor is further configured to display amark when an actual imaging state corresponds to the display positionand the orientation, the mark having a shape and a size identical to ashape and a size of the instruction plane.
 13. The remote support systemaccording to claim 5, wherein the polyhedron includes the instructionplane and further includes a columnar image reaching the target object.14. The remote support system according to claim 13, wherein thecolumnar image is a cylindrical image.
 15. The remote support systemaccording to claim 1, wherein the first processor is further configuredto change a display state of the guidance image in accordance with adeviation amount between an actual imaging state and an imaging statecorresponding the display position and the orientation.
 16. The remotesupport system according to claim 15, wherein the display state is atleast one of a blinking interval, a display color, and a transmittance.17. The remote support system according to claim 16, wherein the firstprocessor is further configured to, when changing the transmittance,increase the transmittance as the deviation amount decreases.
 18. Theremote support system according to claim 1, wherein the first processoris further configured to, when a target position of the target object isimaged, virtually display a marker at the target position.
 19. Aterminal device comprising a processor configured to: transmit, to aremote device, image information obtained by imaging a space including atarget object; receive display information that is derived by the remotedevice in accordance with the image information, the display informationindicating a display position and an orientation of a guidance image tobe virtually displayed in a three-dimensional space in an imaging areafor the target object; and display the guidance image by using thedisplay information that is received.
 20. A remote device comprising aprocessor configured to: receive, from a terminal device, imageinformation obtained by imaging a space including a target object;derive display information in accordance with the image information thatis received, the display information indicating a display position andan orientation of a guidance image to be virtually displayed in athree-dimensional space in an imaging area for the target object; andtransmit, to the terminal device, the display information that isderived.